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i. THE EVALUATIVE QUESTIONS 


Evaluation research, as with all other research, is supposed 
to provide answers to specified questions. The adequacy of the research 
results is dependent, to a considerable extent, on the adequacy of the 


questions which generated the research. 


The most obvious evaluative question is - "Was the program 
successful?", This is a good place to start but considerable refine- 
ment is needed before an adequate research question is yielded. 
Firstly, success rarely comes on a straight yes/no basis - it is a 
matter of degree. Our first revision, then, leads to the following - 


"To what extent was the program successful?". 


The answer to this question, while helpful, is of limited 
utility to other practitioners in the field. Imagine, for example, a 
community worker in Calgary who learns that a very successful anti- 
vandalism program is running in the east end of Halifax. To develop the 
program in Calgary, there are at least two additional pieces of required 


information before he or she can begin developing the program in Calgary. 


1. What was the setting of the program? Was the target 
commmity similar to the community which needs an anti- 


vandalism program in Calgary? 


2. What was the precise nature of the program? What actions 
were undertaken and which of these actions appeared to be 


critical in helping the program achieve its results? 


As a simplistic example of the second point, suppose the 
Halifax group had attempted two concurrent approaches to combat 
vandalism - the development of a citizen group to increase the public's 
awareness of the problem and the development of recreational programs 


for youth. Suppose, further, that because of pre-existing factions 


Digitized by the Internet Archive 
in 2023 with funding from 
Hamilton Public Library 


https://archive.org/details/evaluationofsoci0Ounse 


within the community, the development of the citizens group was a dismal 
failure although the recreational programs took off with a bang. The 
subsequent evaluation shows that vandalism decreased significantly during 
the period of program operation. Apparently, therefore, the recreational 
programs were the critical element in reducing vandalism - not the 
development of a citizens group. It is important that the worker in 


Calgary know about this. 


This, then, represents an evaluation of the actions undertaken 


- of the "process" of the program. Not surprisingly, this is often 


called the process evaluation, 


The evaluation of program success, discussed previously, is 


often referred to as the outcome evaluation. 


The description of the setting of the program is the community 
profile. 


The importance of differentiating between the process and 
outcome components of the evaluation cannot be overstated. Let's return 
to our beleaguered community worker in Calgary. After reading about the 
Halifax project, a report is received which depicts an anti-vandalism 
program in Vancouver which involved the development of citizens groups 
and recreational programs. This program was an abysmal failure. The 
worker corresponds with their counterpart in Vancouver and, after much 
digging, discovers that after one year of operation, only two people 
were interested in the citizens group and three teenagers had turned out 


for the much planned recreational programs. 


The program obviously failed in Vancouver because it was never 


implemented. The outcome evaluation was an evaluation of a non-event. 


The extent to which the program is implemented is a component 
of the process evaluation. Remember, then,to differentiate between the 
evaluation of outcome and the evaluation of process and make sure that 


both are covered in your overall program evaluation, 
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The important point to keep in mind here is that the key 
audience for your evaluation is your colleagues. If social development 
programs are to advance, they need the kind of knowledge base that 
informs all of us about what sort of interventions appear to have the 
desirable effect in given situations - given a particular problem, what 


appears to work and what appears not to work. 


The primary purpose of your evaluation, then, is to produce 
the kind of information that your colleagues need if they are going to 


attempt the same kind of intervention that you have attempted. 
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II, THE EVALUATION MODEL 


The evaluation model which is described in this paper is based 


upon the expressed goals and objectives for the program. This is the 


only meaningful way of evaluating programs of a social development nature, 


This focus upon goals and objectives is of particular importance, given 


the usual lack of attention that they receive. A significant portion of 


this model, therefore, is concerned with the development of an adequate 


set of goals and objectives. 


This serves to illustrate the powerful interdependence of 
planning and evaluation. These two activities are almost completely 


inter-related ~ it is impossible to carry out one without the other. 


This model also serves to focus the attention away from the 
old "red herring" of quantification, The usual reaction to social 
development evaluation concerns the apparent impossibility of quantify- 


ing the effects of social development programs. 


The past use of this model has demonstrated time and again 
that, when evaluation is preceded by adequate planning, the quantifica~ 
tion of outcome is the simplest part of the whole process. As the 


planning proceeds, quantitative indicators almost suggest themselves. 


The essential steps in applying this model are as follows: 


1) Clarify program goals and objectives, starting with outcome 


goals, through the development of a goal hierarchy; 
2) Priorize goals in the hierarchy; 
3) Identify quantitative indicators of outcome; 
4.) Develop program plans or action plens using goal indicators; 
5) Develop activity objectives; 
6) Quantify activity objectives; 


7) Identify required resources and possible measurement. 
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The actual evaluation report should address the following 


issues: 


1) What was accomplished - i.e. what were the outcomes and their 


priority (Steps 1, 2, 3 above); 


2) What was done to accomplish this - i.e, what activities were 
undertaken and which ones attained their goals (Steps 4, 5, 
6 above); 


3) What resources were required to accomplish this (Step 7). 


The important point to remember here is that the most important 
audience for your evaluation is your colleagues. The question to con- 
stantly keep in mind is - "What information do my counterparts in other 
areas need if they are planning on dealing with the same problem or 


issue?", It is this information which is of key importance, 
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ITI. DEVELOPING PROGRAM GOALS AND OBJECTIVES 


The most important phase of evaluating a program, whether 
preventative or rehabilitative, is the formulation of program goals. 
They serve as the benchmark against which the success of the program 
is measured. Unfortunately, however, they rarely receive adequate 


attention in program planning. 


A concern with goals and objectives is not unique to the 
community service environment. It has also been a long-time pre- 
occupation of the private sector, culminating in the present day 
administrative approaches of "management by objectives" and "management 
by results", 


‘ 
William Newman (1950) in one of, the ‘classical guides on 
business administration summed up pegerie. of goals as follows: 


"Distinctive benefits se Garret up goals include: 
(1) Purposeful and integrated planning is made 
easier. (2) Unproductive work is more likely to 
be avoided, (3) ating goals, or standards, 
can be used i building b 


blocks in developing 


programs. (4) Goals also serve as standards for 


va sie 


purposes of control, and they play an important 
part in motivation. (pg 21-28) 


The early writing of Peter \ and E.C. Schleh 
(1961) have been refined into fairly comprehensive systems of management 


by objectives and results by writers such as Karl Albrecht (1978). 


The criteria for writing effective goal statements as set out 


by Albrecht are presented in Table I, 
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TABLE I: THE CHARACTERISTICS OF AN EFFECTIVE GOAL STATEMENT 
(reproduced from Albrecht (1978) pg 75-76) 


SPECIFIC - It spells out, in concrete terms, what is to be achieved, to 
what degree, and the deadline for achieving it. It uses very specific 
terms instead of abstractions. It focuses on performance, and it uses 
performance variables whenever possible. It tells how much, how many, 
how big, how small, etc. Or, it identifies specific qualitative 
conditions which leave no doubt as to their attainment. 


PAYOFr-ORIENTED - It identifies a set of end conditions which have 
intrinsic value, or which are associated with something of value. It 
is clearly a worthwhile thing to achieve; there is an unquestionable 
element of value in the conditions it identifies. 


INTRINSICALLY REWARDING - It specifies a desired set of conditions 
which will bring rewards to the person who is to strive for them. That 
is, there must be associated with the goal a reward which the "action 
person" wants and is willing to work for. This may be simply a mtter 
of recognition accorded the action person for achieving it; or it may 
be a payoff associated directly with the goal. But, in any case, it is 
only a goal in the eyes of the action person if he himself can foresee 
a personal payoff in accomplishing it. 


REALISTIC — It identifies a target which is reasonably attainable. It 
accounts for practical experience and various uncertainties, and it 
allows a reasonable margin for error, It rests upon reasonable assump- 
tions about the future and about the people who will work to achieve 
the goal. 


OBSERVABLE - It specifies a set of conditions which can be detected, a 
target which can be identified to the satisfaction of all concerned, 
especially the people responsible for its achievement, These conditions 
will be clearly recognizable when the goal has been achieved, 
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MISSION STATEMENTS, GOALS, OBJECTIVES AND ACTION PLANS 


Albrecht uses, as a starting point in his discussion of goals, 
the concept of "fuzzies" which are simply vague statements of intent 
which are not specific enough to allow for a recognition of whether or 
not it has been achieved. In community services, these phrases are often 
called "mission statements" or, in a less complimentary vein, "motherhood 


statements", 


The family service mission of "strength to families" would be 
a community service example. In community-based preventative programming, 
a statement such as "the improvement of the quality of life in the Baxter 
neighbourhood" would fall into this category, In Albrecht's schema, the 
mission statements are the necessary starting point in any process of 
goal definition, This should be followed by the development of more 


specific goals and action plans for their attainment. 


There is considerable semantic confusion around the terms 
"goals" and "objectives" in terms of their relative specificity. Some 
writers use goals as the more specific of the two, while others use 
objectives. As an example of the former, Albrecht's schema posits that 
objectives can be either "fuzzies" or "goals" with goals being the 


specific, operational statement of the objective. 


Community service applications, however, have tended to use the 
opposite approach whereby objectives are the measurable translation of 


goal statement:- 


",,.an objective can be either a lower level of 
goal indicator accomplishment of a vital 
preliminary pre-requisite to the attainment of 
the goal. (Grabowski, 1978, pg 38)" 


"The primary characteristic, that defines an 
objective is that it can be stated in operational 
terms. By this we mean that the objective can be 
stated in such a way that the extent to which it 
is accomplished by programs can be measured or 
assessed. (Drezner and McCurdy, 1979, pg 38)" 
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An objective then, as defined above, 
quantifiable and measurable version of a goal. 


ization which will be utilized in this paper. 


is simply a more specific, 


This is the conceptual- 


THE HIERARCHY GOALS 


Goals and objectives can be arranged in a hierarchy based upon 
their specificity such as illustrated in Figure II. Goals A, B, and C 
are more specific versions of the mission statement. Sub-goals are more 


specific yet and objectives are the measurable expressions of intent. 


FIGURE II: A HIERARCHY OF GOALS 


Mission 
Statement 
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Suppose, for example, that one of the goals of a community based 
anti-vandalism program is to increase community awareness about vandalism. 
The two sub-goals may be a citizens committee and a public education 
program. The objectives for the citizens committee can then be framed 
around time (the schedule for setting up the committee); attendance of 
members; the satisfactions of committee members and committee effective- 


ness. 


When a hierarchy such as this is set up, there are some impor- 
tant criteria to be used in formulating sub-goals (Drezner & McCurdy, 
1979). These are as follows: 


ASSUMPTION 1. The complete set of subgoals in any branch should 
be collectively exhaustive. In other words, no other subgoals 
need to be considered and the achievement of all subgoals should 
completely fulfill the goal from which they derive. 


ASSUMPTION 2. The achievement of each subgoal should make a 
Significant and valid contribution to the goal from which it 
derives. The contribution should be in tune with the spirit and 
intention of that goal. 


In other words, the step of identifying component subgoals should 
not simply be an exercise in logical dissection. 


ASSUMPTION 3. Each subgoal should be independent of other sub- 
goals, In other words, the value of its achievement should be 
unaffected by the achievement of other subgoals. 


ASSUMPTION 4. The achievement of a subgoal should contribute 
only to the achievement of the goal from which it is derived and, 
through that goal, up the tree in a direct line to the high-level 
goal at the top, It should not contribute to any goals that ere 
not in this direct line. 


ASSUMPTION 5, The goals and subgoals that make up the goal 
structure should be feasible. High-level goals and those sub- 
goals near the top of the structure may carry some uncertainty 
when they could be achieved, Lower-level goals and especially 
those qualifying as objectives, however, should be feasible to 
some degree in the planning horizon. (pg 41-42) 
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PROCESS vs OUTCOMES 


There is also some confusion about process and outcome goals. 
Process goals are statements of some kind of planned activity, i.e. 
the provision of counselling services; the development of recreational 
programs, etc. Outcome goals, on the other hand, are the statements of 
the expected results of the activity, i.e. "counselling success rates"; 


number of participants; satisfaction of participants, etc, 


This can be conceptualized as a distinction between program 
inputs and outputs (Grabowski, 1978). "Outputs are things sought, inputs 
are resources used to achieve the outputs.” (pg 37) Grabowski seems 
adamant that goals should always reflect outcome, As a means for assur- 
ing this, he suggests that goal statements should be written without 


verbs to make certain that it is not an activity (input) statement, 


From an evaluative point of view, the emphasis should be on 
outcomes, There is, however, a definite role for process goals, 
particularly from the point of view of program planning, If an outcome 
goal is achieved, it is important to know which activities or elements 
of the process seemed to be most important in terms of achieving that 
goal, The same was obviously true of an outcome which was not achieved. 
For example, suppose there were three activity or process goals 
associated with a specific outcome goal, The outcome goal is achieved 
but one of the process goals is not. Apparently, therefore, that 
particular activity was not very important in terms of the outcome goal. 


That is important information for program planners, 


In summary, then, outcome goals are the basis for evaluation, 
Process goals, however, are very important to the interpretation of the 
outcome results. As a simple rule of thumb, begin with two questions - 
"What is the program attempting to accomplish?", and "How is it going 
to accomplish these things?", The answer to the first question leads 
to the outcome goals, while the answer to the second leads to the 


process goals. 
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DEVELOPING THE HIERARCHY 


The hierarchy is best developed in a "brainstorming" kind of 


A small group of perhaps 6-12 persons who are knowledgeable 


about the area of work covered by the starting goal. As you work your 


way through the hierarchy, the goal should become more and more specific 


until, at the lowest level, they represent quantitative indicators of 


outcome, 


The general steps in working through this process are as 
follows: 


1) Given your first general goal statement, ask "How will we 
know if this goal is being accomplished?" or "What kind 
of things should we look for to see if we are making some 


progress in this area?" 


2) In a very unstructured and "free-floating" approach, the 
group will then start throwing out ideas which are listed 
on a flip chart. Participants should feel free to throw 
out any ideas that occur to them, wntil their ideas are 
exhausted, The elements listed here should be somewhat 
more specific than the general goal but do not worry if 


they are not yet quantifiable. 


3) The list is then reviewed to form "clusters" of ideas that 
appear to be interrelated or all dealing with "different 
aspects of the same thing". This is a fairly intuitive 
process which requires some discussion about each item on 
the list. Some may be discarded because they appear to be 


activity or process statements rather than outcomes. 


4) Give each of these clusters a goal name, which seems to 
summarize their common intent. These goals then constitute 


the next level of the hierarchy. 
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5) Take one of those sub-goals or clusters and, on another 
flip chart or blackboard, list the elements from the 


original list which were relevant to that cluster. 


6) Then, begin the process again by asking, "How will we know 
when this goal is being accomplished?" and "What kind of 
things should we look for to see if we are making some 
progress in this area?", Repeat steps 1 to 6 until you 
have arrived at specific, quantifiable indicators of out- 


COME « 


It is through this kind of process that you evolve from a very 
general statement such as "To increase the participation of women in the 
community" to "x number of women serving, in a meaningful fashion, on 
voluntary Boards in the city". The "meaningful" aspect of this goal can 
be measured by distributing questionnaires to women involved on the 


various Boards. 


AN APPLICATION TO PREVENTATIVE AND SOCIAL DEVELOPMENT PROGRAMS 


Consider the following general statement of program purposes: 


"To develop, through a committee of local 
citizens, a series of youth recreational 
programs to decrease vandalism to the city 
average in the Lansdale community." 


The goal statement would read: 


A vandalism rate equal to the city average. 


The reference to the citizen groups and the recreational 
programs describes the actions which are planned in order to achieve a 


reduced vandalism rate. 
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If this sounds rather narrow-minded to the reader, that is 
because it is. This introduces a unique element of preventative 
programs - some of the means are also ends, The involvement of 


citizens in solving community problems is a benefit in and of itself, 


In formulating goals for preventative programs, we must 
therefore differentiate between participant benefits and community 
benefits. In the previous example, the community benefits involve 
decreased vandalism. The participant benefits, particularly within the 
citizens committee, are related to the involvement in solving community 


problems. 


The problem, here, was in the original formulation of the 
purpose, which was written in such a way as to suggest that, in fact, 
the community involvement was only a means for decreasing vandalism. 
This is not an unusual problem, Authors of preventative program pro- 
posals often combine the elements of participant and community benefits 
into one statement of purpose in which participant benefits become 


relegated to a "means" role. 


Begin, then, with two statements of purpose - one relating to 
participant benefits and the other relating to community benefits. 
This will result in separate goal statements for the participants and 


the community. 


If we apply this to the original example, we end up with three 


goal statements :- 


1. An involved committee of citizens; 
2. A series of recreational programs; 


3, A vandalism rate equal to the city average. 
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There is one further test to apply in testing the adequacy of 
your goals - ask why. If the answer is a general motherhood type of 
statement, then you probably have an adequate goal. If, however, the 


answer is more specific, your goal may yet need some refining. 


Why do you want a series of recreational programs? If the 
answer is to keep the kids off the streets so they will not commit 
vandalism, then you do not have a goal; you have an action statement. 
Drop it as a goal. If, on the other hand, the answer is that recrea- 
tional programs are good for kids because of the camaraderie and 
character-building aspects of the program, then we are back to a 


motherhood statement and it is probably an adequate goal statement. 


At the next level of specificity, the objectives for the goal 


relating to "an involved group of citizens" could be - 


1. Participation of "x" community representatives (from 
different sectors of the community). 


2. Attendance rate of "x" 
3. Participant satisfaction rate of "x" 


4. Attainment of committee goals. 


As suggested by number "4" above, the committee, in turn, may 
develop goals and objectives of its own which can then become the next 


level in the hierarchy. 


The same process should be applied to the other two goals. 


Leadership Skill Development 

A more complex example, in a social development sense, would 
be one dealing with "leadership development in indigenous group". Given 
that starting goal, the following list was generated in a brainstorming 
session as a result of the question - "How will we know when this is 


being accomplished?", 
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1) Increased independence of the group, i.e. group will 


require less "nurturing" by development workers. 


2) Better planning process in place, i.e. clearly defined 


goals based on group consensus, 
3) More involvement in other agencies, 


4) Increased number of people involved in decisions within 


the group. 
5) Improved accessing of resources. 
6) Increased activity in group. 


7) Increased number of group members fulfilling leadership 


roles outside of the group. 
8) More output. 
9) Higher profile in the community. 


10) Improved perception of group by community and other 


groups. 


11) Increased credibility with significant others, 


From this listing, the following four clusters were identified - 
improved internal functioning; improved output; improved community 
profile; improved involvement with other groups and issues, The next 


level of the hierarchy, then, appeared as follows: 


LEADERSHIP DEVELOPMENT 
IN INDIGENOUS GROUPS 


Improved Improved Improved Networking 


Internal Output Community 
Functioning Profile 
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A repeat of the eession with the community profile goal 
produced the following listing: 


1) Media visibility - number of mentions in the newspaper, 


number of radio talk shows, etc. 
2) Number of communications with other groups. 


3) Number of group members involved on Boards and committees 


of other organizations. 


4) Number of decision-making bodies including formal partici- 


pation of the group. 
5) Number of meetings between group and politicians. 
6) Number of new Board members on group. 


7) Number of group members. 


This process, then, quickly yielded some powerful quantitative 
indicators of outcome. The lists above are used as examples and are, 


by no means, exhaustive. Other elements could certainly be added. 


Assuming that similar lists have been generated for the other 
three components of the hierarchy, it is then importan* to priorize all 
of these outcomes, Obviously, any program would be hard pressed to 


accomplish every one of the outcomes within a given time frame, 


The next three sections of this guide deal with alternative 
methods for carrying out the priorization based upon how many people are 


involved, 


Given the high priority outcomes, it is then necessary to focus 
on activity goals - what activities must be undertaken to achieve these 
outcomes and how will we know when those activities have been undertaken 


and/or completed. This, then, is the process evaluation. 
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Specific indicators for the activity completion can be 
identified and monitored. This allows you to answer the question - 
"Which activities appeared to be important in achieving or not achieving 


this outcome?", 
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IV: GOAL PRIORIZATION: INDIVIDUAL APPROACH 


The priorization of goals is an important part of any goal- 
based evaluation, It is rare that all of the goals which are developed 
will be of equal importance. The attainment of some goals will usually 
be more important than the attainment of others and some kind of 
numerical weighting system should be utilized to reflect these differ- 


ences. 


The simplest method is to simply sit down and rate each goal 
in terms of its importance, This rating can be carried out on any kind 
of numerical scale, i.e. 0 to 100; O to 103; O-5. The upper value 
represents the greatest importance score while the lowest value 


represents the least important. 


In their book, "A Planning Guide for Voluntary Human Service 
Delivery Agencies", Drezner and McCurdy (1979) use a 100 point scale 


as follows: 


"STEP 1. Rank the high-level goals in the order of their 
importance as you view them (first for the most important, 
second for the next, and so on). Assign the value of 100 
to the high-level goal you ranked first (that is, the most 
important). There is no 'magic' to 100; it just seems 
easier for many people to deal with. Then rate the other 
high-level goals relative to this most important goal, If 
the second-ranked goal seems about half as important, rate 
it 50. If the third is just a little less important than 
the second, you might rate it 45, and so on. Try to use 
as much of the scale (0-100) as you can; do not, for 
example, group all high-level goals between 90 and 100 if 
you can help it, 


When you have a rating for each high-level goal, 
review them to see if they reflect your opinions, If one 
goal is rated 60 and another 30, does the second seem about 
half as important as the first? If not, change the ratings 
until they better express your views. Note the form of the 
question: How much more (or less) is the value of one goal 
relative to another? Or, of course, are they of equal value? 
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This form is important because it helps get a valid 
expression of feelings that can be used later when perform- 
ing arithmetical operations. It is convenient to keep the 
most important high-level goal rated at 100 and to change 
the ratings of the other goals to reflect their importance 
relative to this high-level goal. (pg. 53)" 


In an equally excellent book, J.B. Ounningham (1978) used a 
weighting system whereby the sum of the weights add to 100. Conceptually, 
this means indicating the importance of each goal as a percentage of the 
importance of all the goals. Thus, if three goals are judged to be 


equally important, then they are all rated 33.3. 


This is also the technique which Drezner and McCurdy (1979) use 


in priorizing the second level goals as follows: 


"STEP 2. When you have the high-level goals rated, pick one 
of these goals and examine the component, second-level goals 
that evolve from it. Express your feelings abovt the 
relative contribution each of these second-levei goals makes 
to the high-level goal in terms of a percentage. In effect, 
you are dividing up the value of the high-level goal among 
the second-level goals, so the percentages should add to 100 
percent. (For example, if there are three second-level 
goals, you might say the first contributes 50 percent, the 
second 40 percent, and the third 10 percent). 


Next, pick a second-level goal that is broken down 
into component third-level goals and repeat the process by 
dividing the value of that goal among its components. 
Continue until you reach the lowest level subgoals (object- 
ives) in each branch, then go through the same procedure 
for the other high-level goals. 


STEP 3, This step involves some relatively simple calcula- 
tions made even simpler if you have a calculator. To see 
how it works, assume a high-level goal has a value of 60 and 
that there are two subgoals below it. Assume the first of 
these two was given 70 percent of the value and the second, 
30 percent. To get the value of the first subgoal, multiply 
60 by 70 percent (and divide by 100) and you get a resulting 
value of 42 for this component goal. Next multiply 60 by 30 
percent (and divide by 100) and you get a resulting value 18. 
You have now computed the values of the two second-level 
goals - 42 and 18, These add up to the value of 60 attached 
to the high-level goal from which they derive. 
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them out in the open and formally recognize their importance, 


STEP 4, This step is an important reminder that the process 
is not Simply an exercise in numbers. Even if you are short 
of time, we recommend that you do not ignore this step. 


The values attached to each subgoal by the process 
described in step three should still convey your preferences 
after the calculations. Look at the values. Do any seem out 
of line? If two lower-level subgoals have nearly the same 
number (value), are they about equally important? 


When you find discrepancies between the numbers and 
your preferences, consider changes to better express your 
intent. Note that you can shift the percentages you used to 
divide up the value of the goal immediately above. (Remember 
that if you raise one percentage, you need to lower one or 
more others to keep the total equal to 100 percent.) Note 
also that you can change the rating (the value) assigned to 
the high-level goal, This will change the values of all sub- 
goals beneath it. 


Keep going with this review and fine tuning wntil 
you feel comfortable with the result as an expression of your 
preferences, Again, this is not supposed to be an exercise 
in numbers, If it were it would have no validity and no 
meaning, It is, therefore, a time-consuming task requiring a 
great deal of concentrated thought. If it is being done by 
a group it also requires a good deal of dialogue and rethink- 
ing. In actual practice, however, many agencies have found 
it necessary, for reasons of time, to accept an adequate 
expression, rather than optimal expression, of preference 
during the initial planning cycle. The result can still be 
of use in clarifying many issues and it can serve as a valid 
base for the remainder of this program planning system, 


(pg. 54-55)" 


USING SPECIFIC CRITERIA 
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The previous ratings have all been quite subjective in nature. 


As you sit down to rate each goal, considerations of costs, expected 
success, effort involved, etc. will probably create considerable 
indecision. Should I rate it according to political attractiveness? 


Ease of success? etc. etc. 


The best way of dealing with these criteria is to simply bring 
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the criteria which you consider important in deciding upon a priority 


rating for the goals. A sample list could read:- 


1) community benefit 

2) cost of the program - the cheaper, the better 

3) chances of success - the higher, the better 

4) political attractiveness - the higher, the better 

5) community readiness - the higher, the better 

6) length of time needed for completion - the shorter, the 
better. 


There are undoubtedly many others which the reader can apply 


to specific situations. 


The criteria are, themselves, of varying importance - some 
are more important than others and it is, therefore, necessary to apply 
some kind of weighting system to them, The method described below is 
an adaptation of the approach described by Drezner and McCurdy (1979). 


Firstly, assign a weight to each of the criteria, using any 
numerical scale, although the authors again used a O to 100 scale, 


Suppose the five, previously identified criteria, are rated as follows: 


1) Cost of Program 60 
2) Chances of Success 80 
3) Political Attractiveness 50 
4) Community Benefit 90 
5) Commmity Readiness 90 
6) Length of Time Required 60 


Secondly, each goal should be weighted against each criteria. 
Again, any weighting scale can be used, but the easiest is a simple 
ranking system. For example, rank the goals according to the antici- 


pated costs of achieving them - staff time, equipment costs, etc. 
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If we are using three goals, the cheapest will be ranked "3" and the 
most costly goal will be ranked "1", 


The other goals are then ranked in the same manner with the 
"3" value corresponding to the most desirable state - ise. cheapest, 


greatest chance of success, greatest community benefit, etc. 


The easiest way to do this is to set up a matrix, as 
exemplified in Figure IIIa. The numbers in the top row are the 
criteria weights and the number in the upper left hand corner of each 


box are the goal weights. 


The third step is to mltiply each pair of goal weights and 
criteria weights and write the product in the appropriate box as show 


in Figure Lilb, 


Bach of these values is then summed for each goal to yield 
an overall priority score for each of the three goals, as in Figure 
gi ic, 


This priority score indicates the relative importance of each 
goal according to that set of criteria. They can be reduced to a more 


manageable size by dividing each score by 10 or 100. 


This method can then be applied to second level goals, if you 
wish. It may, however, simply be easier to use the method discussed 


earlier for these goals and objectives. 


Up to this point, the methods of goal priorization which have 
been discussed have been carried out on an individual basis. You may, 
however, want to use a group of persons to carry out the priorization, 
The use of a group is, of course, somewhat more complicated but there 
are standardized procedures which can simplify the proceedings. These 


procedures will be discussed in the next section. 
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FIGURE IIT: GOAL/CRITERION MATRIX 
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V: THE DELPHI METHOD 


One of the most popular structured decision-making procedures 


is the Delphi method, originally developed by the RAND Corporation in 


the United States. It is a method for structuring group decisions 


through voting, feedback and dialogue with the main aim of arriving at 


consensus. As with any general approach, it comes in a variety of dif- 


ferent forms and the one discussed here is one of the simpler approaches. 


In terms of its application to assigning priority ratings to 


goals, the steps in implementing a Delphi process are as follows: 


1 


2) 


3) 


4) 


5) 


Kach group member would privately rate the priority of each 
goal on some standard numerical scale, such as the O to 100 
scale utilized in the previous chapter, In this presenta- 
tion, a simple 1 to 10 rating scale will be utilized where 
"1" represents very low priority and "10" represents highest 


priority. 


Their responses are then turned into a co-ordinator who plots 
the responses on a graph, such as the one presented in Figure 
IV. The members do not identify themselves on the form which 


they turn in so their confidentiality is assured. 


The graph is then displayed to the group and discussion 
follows. Individuals are free to argue for the various 


priority ratings. 


Step 1 is then repeated with the members again rating each 
priority in light of the discussion, The results are then 


turned in once more, graphed and the graph is displayed. 


This process continues until either consensus is reached or 


there is no change in the graph after two or three tries. 
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GRAPHING DELPHI RESULTS 


FIGURE IV: 
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INTERPRETING THE GRAPH 


This graph is one of the easiest ways of displaying and 
measuring consensus, There are more sophisticated statistical methods 
for measuring the actual amount of consensus in a group but for the 


purpose of the present utilization, the graph works fine. 


Figure IVa shows the graph of a group which has arrived at a 
fair degree of consensus, There is not a total agreement but, then, 
in any heterogeneous group, it is probably not reasonable to expect 
total consensus, For the overall group rating, the individual 


responses are averaged, 


Figure IVb shows the responses of a hypothetical group which 
contains two polarized factions. If, in such a situation, the two 
groups do not come any closer together after more discussion, your only 
alternative is to take the average rating which will be somewhere in 


between the two groups. 


Figure IVc shows a situation where there is one individual 
in the group who refuses to be moved by the opinion of the rest of the 
group. If this arises, you can take the average of the group or it is 


also acceptable to simply drop the extreme value, 


USING CRITERIA 


Up to this point, the procedure has involved the group in the 
subjective weighting of the goals, If there is a perceived need for 
some criteria to be used, as described in the previous chapter, then 
the Delphi has to be used twice - once to arrive at the criteria 


ratings and once to arrive at the goal ratings. 


The average group criteria and goal ratings are then combined, 
as described in the preceding chapter, to arrive at the priority scores 


for the goals. 
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USING SECOND LEVEL GOALS AND OBJECTIVES 


This process can also be applied to second level goals and 
objectives, as described previously. In this situation, it is best 
to use the "percentage scale" whereby values ranging from 0 to 100 are 
applied to each second level goal or objective such that they sum to 


100 (see preceding chapter). 


GROUP SIZE 


There is no ideal group size for using the Delphi method. 
However, as the group gets larger, discussion sometimes becomes un- 


wieldy and smaller sub-groups should be formed. 


The Delphi does assume that you can get all the necessary 
people together into some kind of meeting area, Sometimes this may 
not be possible or the really ambitious program planner may want to 
solicit community opinion on a scale which is larger than can be 
comfortably applied to the Delphi approach, If this is the case, a 
questionnaire-based approach, which is described in the next chapter, 


can be utilized. 
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VI: SOLICITING COMMUNITY INPUT 


J. Barton Cunningham (1978) has written an interesting 
publication on the use of an evaluative technique called "Functional 
Goal Evaluation", The method contains procedures for soliciting 


general community responses for program goals and their attainment. 


The steps in utilizing this procedure for priorizing goals 


and objectives are as follows: 


1) Identify the groups within the community from whom a 
response is required. Cunningham was concerned with 
police programs and his groups were high school 
students, juveniles in conflict with the law, R.C.M.P. 


and community at large. 


2) Construct a questionnaire which presents the program 
goals and requires the respondents to rate these goals 
in terms of their importance on a percentage scale. 
Second level goals can also be included and rated in 


the same fashion, 


Cunningham employed six general goal areas - crime prevention; 
criminal investigation; traffic control; community education; 
administrative co-ordination and efficiency; public image. As indicated 
above, all of the respondents rated the importance of these goals on a 
percentage scale such that the total weights of all the goals summed to 
100. 


At the second level, Cunningham employed a number of dimensions, 
many of which were activity statements rather than goals. As an example, 
the following sub-categories were used for the crime prevention goal 
area - working with and helping teenage offenders; developing positive 
feelings with students in the school system; skillfully dealing with 
family problems and disputes; developing police/student sports 
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activities; ability to refer problems away from court; the R.C.M.P.'s 
ability to lower criminal offences and vandalism, These six dimensions 


were also rated on a percentage scale, 


One of the advantages of this method lies in its ability to 
yield data about the sometimes conflicting expectations of different 


groups within the community. 


For individuals who are interested in pursuing this approach, 
the Cunningham publication includes the work sheets and forms which are 


needed for carrying out the process, 
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Ministry of the Solicitor General, Ottawa, 1978. 
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VII: SCALING GOAL ATTAINMENT 


It is unlikely that, in most circumstances, goal attainment 
can be measured on a simple yes/no basis, It is usually helpful to 
think in terms of "levels" of goal attainment and, correspondingly, 
"levels" of program success. The method which can be utilized in this 
situation is goal attainment scaling (Kiresuk and Sherman, 1968). This 
method, which was originally designed for use in clinical evaluations 
is sufficiently general in nature to be applied to almost any setting. 


It applies equally well to outcome and process objectives. 
A sample goal attainment scale is presented in Figure V. 


The approach is fairly simple. Each goal is re-written in 
terms of levels of attainment, with the middle level being the expected 
outcome, In the example of Figure V, a vandalism goal has been utilized 
and scaled to five levels based upon the city average vandalism rates, 


The outcomes are scored from -2 to +2 as indicated. 


Figure VI shows a worked example for a series of sub-goals 
under a general "Formation of a Committee Goal", Each of the four sub- 
goals has been weighted according to importance, For the sake of 
simplicity, a weight of five was attached to goals 3 and 4, while goals 
1 and 2 were given a weighting "3", In fact, any of the previously 


described methods could have been utilized. 


The last goal, "committee goals", would relate to a further 
set of goal attainment scales which would have been developed to 
measure the outcome of the committee activities. That score would be 


transferred to this form as part of the overall evaluation. 
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FIGURE V: A GOAL ATTAINMENT SCALE 


eae To reduce the rate of vandalism in the Lansdale 
* neighbourhood to the City average 


Outcome Indicator 
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expected results 
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Moderately less 
than expected results 


Predicted or 
expected results 


Vandalism equal to 
City average + 5% 


Vandalism rate 5% to 
10% less than City 
average 


Moderately more 
than expected results 


Vandalism rate more 
than 10% lower than 
the City average 


Much more than 
expected results 


ae) 


— Sohsteedinie ne 


Fibacul odd, at natiabiw 20. 232n 


‘ 
toFsorchal “T945 ay 


ake __ ee 
Poy4é #1. Di 7 ' 

svods RL iT == Stor ‘~ 
wave gto ] 


to te i ae - a a) i | 
ed Re simr oert bres - enet Uaterabolt t 
FEO saul tiimen Botoegee: nai) } 


wa | paarepere =e 
etee etiuuen Datpegxe | | 


; ‘ : SS 
tf Ke ofet metiabaav | atom yisjarsbo 7 
lo neds amel ROL | LS | avliveet Beduegke neds | 


HT cA 
myer y¥ a2 


<i S © 


oe _—— —— a 


sent oon. dou 
ET do Jasper 


bevel 


mou 


(G=00ue4 1OduT ) (G=e0Ue71odut ) 
seatqzoatag 
904.94 THMOg | go uoTqyoesIstzes 


Te* = 
t 9OT=S+S6+E+€ = O2009 dd0UeZIZ0dNT TRIO] 
a 81009 SVD 
G 225 OF {e7) ce 6 = ef00g suloo4yNp Teo], 
§ = (S)  (T) | O=(§*0) | € = (E) (TH) | € = (E) * (TH) ex00g owtooaT0 | 
| —_— 
| a w+ | 
cr sy[Tnset 
A 
gO | %T6 + noe Le peyzoedxa 
eI00g SYD + ueyy srow yony 
| t+ 
T+ uPoTgsTyes Area, ee 
peqoedxa 
Fo 04 %06-TS AT Woon 
@I00G Sy Tenbs guryeil osei1eay ee ae 
ovo ‘ ATezerepoyl | 
| 
O uPETISTPESy, | Souepuey ye ° SyTnses 
go oF #08- LL ATH9ON-Tq peyooedxa 
aI009 SYD Tenbs Suryer odereay JO odeIOAY IO po,otperg 
| 
5 lay peTfstzesstp jie pee 
ib ° . Ci poqoedxe 
go 04 AT yy UOW 
81009 SYD Tenbe Zurzer esei1oay ee ae a 
| : ATe eLEpoW 
- | 
oe uPeTtsstpesstp Aton, s}[nset 
go 04 405 - Beh ee peyoodxsa 
81009 SYyy Tenbs surqyer oseroay ueuy SSeT ony 


(€=90Uez10duUT ) 
SIOQUSW, 
gO 9o0uepued1y 


(€=90Ue toduT ) 
SSUTYOOHI 
go s£ouenbar,7 


SIOquoy] 


SHIVOS LNGWNIVLEVY Tvod 40 LHS Vo -IA qunold 


nhaitett, neath yise" 


a 


aoe AD fetme ortiter operas | 
m6 ed 
f. hs ftartacath" 


vistetshoM 
as ened 


eee ——— — = = - is 
4 — 5 ° + a ‘ - 
F ia) 15) i? x < it. { cs 
aiiccsee a a —_ <j —— Ss — 
3 f - $ ’ 
5 - oP nr : : { 4 { ~\ r “Tf 13 u 
= WwH5G, Gin) a = 
- 
f i - = _ r de | 
~ i. o- - 


BE = 


=o. 


SCORING THE SCALES 


One of the advantages of this system is that it allows for the 
calculation of a numerical, goal attainment score, This score is calcu- 
lated as follows: 


1) For each scale, multiply the outcome score +2 to -2 by 
the weight; 


2) Add these products across the scales to calculate a total 


score (pay attention to the positive and negative signs); 
3) Divide these numbers by the sum of the weights used; 


4) The resulting goal attainment score should range, again, 
from +2 to -2. This indicates the overall level of out- 


come for that particular set of goals, 


These calculations are shown for the example in Figure VI. 
By this method, the resulting score can range anywhere from +2 to -2. 
The score of .31 indicates an outcome which was slightly better than 
expected. (The hypothetical outcomes in this example are shown by the 


check marks, ) 


This method, then, yields a simple quantitative indicator of 
goal attainment. Each goal and set of sub-goals can be scaled in the 
same manner yielding a goal attainment structure such as the one pre- 


sented in Figure VII. 


In this hypothetical example, Goal A had a positive outcome 
which was somewhat better than expected. Moving to the sub-goals, this 


seems to be largely due to the success of sub-—goal A-1l. 


Goal B had an outcome which was consistent with expectations 
of the planners, At the sub-goal level, however, the outcome of Goal B2 
was less than expected. Apparently, however, that particular sub-goal 
was not particularly important in the attainment of the overall program 


goal. 
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FIGURE VII: HYPOTHETICAL GOAL ATTAINMENT STRUCTURE 
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The outcome of Goal © did not live up to the planners 
expectations, apparently as a result of a failure to achieve sub-goal 
Cl. Sub-goal C2, however, seems to have been a raving success. 
Apparently, therefore, the attainment of Cl was a more important pre- 


requisite for the attainment of Goal C, than was sub-goal C2, 


If this structure was continued to another level of sub-goals 
or objectives, then it might be possible to find out what went wrong 


with sub-goals B2 and Ol. 
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